The present technology relates to an image processing device, an image processing method, and a program. Specifically, it enables to optimize a search range when calculating a motion vector between images picked up continuously.
In the past, generation of one sheet of a wide angle image (panoramic image) is carried out by linking, for example, strip images cut out of a plurality of sheets of picked up images obtained by carrying out image pickup while moving an image pickup direction of an image pickup device. A setup of a cut out region is carried out, by calculating a motion vector between picked up images, so as to make images in a linkage portion as a continuous image based on the calculated motion vector.
FIG. 1 illustrates a behavior in the past, and shows a positional relationship of four sheets of picked up images from time t−3 to time t obtained by carrying out image pickup while moving in an image pickup direction of an image pickup device.
In a case of obtaining an inter-image motion vector (hereinafter, referred to as a global motion vector) between an image at time t−1 (reference image) and an image at time t (target image), an equidistant range up, down, left, and right from a center of the reference image is defined as a search range. By judging a degree of matching with the reference image in this search range, a motion vector in a location of a highest degree of matching is defined as a global motion vector. In calculation of the global motion vector, in a case of directly matching the entire target image with the entire reference image, a matching image region is only in a part of the images and it is difficult to accurately calculate the global motion vector. Accordingly, as in Japanese Unexamined Patent Application Publication No. 2009-219082, for example, calculation of a global motion vector is carried out by dividing an image into several small blocks, and after calculating a motion vector (hereinafter, referred to as a local motion vector) for each block, by processing them integrally.
FIGS. 2A, 2B, and 2C illustrate a relationship of a block and a local motion vector. A positional relationship shown in FIG. 2A represents a relationship of a target image and a reference image on the basis of absolute coordinates, and FIGS. 2B and 2C represent it on the basis of images.
When target blocks to calculate a local motion vector are set up on a target image, in a related art, a search range is provided that is equidistant up, down, left, and right from an identical position on the reference image as a center. In this search range, matching of the target blocks with the reference image is carried out in series, and relative coordinates of an optimal matching position, which is a most highly correlated position, and a position of a target block shows a local motion vector.
The target blocks may be set up as each of the regions into which the entire screen is divided simply at equal intervals, and may also be as divided regions allowing an overlap. In addition, feature points may also be extracted from edge information and the like in advance to define only a limited area including them as designated blocks. Even when setting up a search range as in FIG. 2C, a region having no image subjected to matching makes no sense. Accordingly, actually, as shown in FIG. 3B, a search range is set up.
The target image and the reference image have consistent images only in a common image region (region shown with hatched lines in FIGS. 2A, 2B, and 2C and FIGS. 3A and 3B). Accordingly, among local motion vectors in all blocks, only those belonging to the common image region are extracted from reliability information, such as a matching degree. Further, by integrally processing one or a plurality of highly reliable local motion vectors thus extracted, a global motion vector is calculated. By shifting the target image based on the global motion vector obtained in such a manner to link it with the reference image, a wide angle image can be generated with the images in a linkage portion to be continuous. As a manner to calculate a global motion vector from local motion vectors, a method of combining, for example, affine conversion and a RANSAC (random sample consensus) algorithm and the like are recognized widely.